feat: support no-evals in perf changelog entries - #2935
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| 选择 eval；若需完全禁用，请在所有相关条目中设置该选项。条目中不能同时使用 | ||
| `all-evals`、`evals-only` 或 `eval-min-prefill-ep`，PR 也不能同时使用两个 eval | ||
| modifier 中的任意一个。这类运行提供吞吐量证据，不提供模型评估证据。 |
There was a problem hiding this comment.
🟡 (optional) The Chinese translation drops the "combined with no-evals" qualifier, so it reads as an absolute ban on using all-evals/evals-only/eval-min-prefill-ep together (or the two PR modifiers together) — which contradicts actual validator behavior (an entry may freely combine evals-only with eval-min-prefill-ep, or all-evals is just mutually exclusive with evals-only at CLI level, absent no-evals). Chinese-only readers get a false validation rule not present in the English page or the code. Fix: retranslate to explicitly tie the prohibition to no-evals, e.g. "与 no-evals 同时使用 all-evals/evals-only/eval-min-prefill-ep，或在 PR 上同时使用 no-evals 与任一 eval PR modifier，都会被拒绝", matching validation.py's validate_append_only_mode and process_changelog.py's no_evals/all_evals/evals_only check.
Extended reasoning...
English (docs/eval-agentx-procedures.md:20-21): "Combining no-evals with all-evals, evals-only, or eval-min-prefill-ep on the entry, or with either eval PR modifier, is rejected" — the ban is conditioned on no-evals, matching infx/matrix/validation.py's model_validator (raises only when self.no_evals is True) and process_changelog.py's main() (raises only when any entry.no_evals and args.all_evals/evals_only). The zh page (lines 18-20) renders this as "条目中不能同时使用 all-evals、evals-only 或 eval-min-prefill-ep，PR 也不能同时使用两个 eval modifier 中的任意一个" — omitting the no-evals connector entirely, so it states an entry can never combine those three fields, and a PR can never use both eval modifiers, neither of which the code actually enforces outside the no-evals case. A Chinese-reading operator following only this page would believe legal configurations (e.g. evals-only + eval-min-prefill-ep) are forbidden.
Verification: nit (docs-only, no runtime break, but introduces false content the base lacks). English docs/eval-agentx-procedures.md:18-20 conditions the rejection on no-evals: "Combining no-evals with all-evals, evals-only, or eval-min-prefill-ep on the entry, or with either eval PR modifier, is rejected." This mirrors infx/matrix/validation.py validate_append_only_mode, which raises ONLY when…
1 participant
Summary
Add opt-in
no-evals: truetoperf-changelog.yamlentries so PR sweeps can run throughput-only validation through the normalfull-sweep-enabledworkflow. This is needed by #2931 to validate Prometheus cache-source attribution without rerunning model-quality evals.Validation
uv run --no-project --with pytest --with pydantic --with pyyaml --python 3.12 pytest utils/test_process_changelog.py utils/matrix_logic/test_validation.py utils/changelog_gate_tests/ -q311 passed. Tests exercise throughput preservation, default behavior, entry isolation, metadata, and conflicting options. No GPU recipe changes in this PR. Prepared with AI assistance.
Note
Medium Risk
Changes PR sweep matrix composition and validation gates; misconfiguration could skip evals unintentionally, though the flag is opt-in and guarded by explicit conflict checks and tests.
Overview
Adds
no-evals: trueonperf-changelog.yamlentries so a normalfull-sweep-enabledPR can run throughput-only for those entries without PR-wideevals-only/--no-evalsCLI tricks.process_changelogstill generates benchmark rows with the existing--no-evalsgenerator path, then skips all eval matrix generation for opted-out entries (same early-exit pattern asappend-only).no-evalsis per-entry: another changelog line can still schedule evals for overlapping configs. The flag is validated against conflicting entry fields (evals-only,all-evals,eval-min-prefill-ep) and against PR modifiers--all-evals/--evals-only.ChangelogEntryininfx/matrix/validation.pydefines the new field (default false). English and Chinese eval procedure docs describe throughput-only sweeps and that these runs are not model-evaluation evidence. Tests cover throughput preservation, metadata, entry isolation, and rejection of conflicting options.Reviewed by Cursor Bugbot for commit a898b0d. Bugbot is set up for automated code reviews on this repo. Configure here.